Repository navigation
Antalya 26.6 Backport of #128217 - Evaluate the PARTITION BY expression of file and object storage writes once per INSERT, not once per block - #2909
Conversation
…tateful-function-spans-chunks Evaluate the `PARTITION BY` expression of file and object storage writes once per INSERT, not once per block Conflicts committed as-is; the next commit resolves them: - src/Storages/IPartitionStrategy.cpp (both modified) - src/Storages/IPartitionStrategy.h (both modified)
Dropped the upstream `getPathForRead`/`getPathForWrite` hunks, which don't exist on antalya-26.6, and kept a one-argument `computePartitionKey(Chunk)` for existing callers.
|
Port-AL: cherry-pick conflicts resolved by AI Conflicted files:
Commits:
|
CI triage for
|
| Check | What failed | Verdict |
|---|---|---|
| Stress test (amd_debug) | Abort on a LIMIT -3 BY query, and the server could not restart because of an Alias→Alias table |
Not this PR (a known upstream issue plus a random stress crash) |
| Stress test (amd_asan_ubsan) | Logical error: 'Not-ready Set ...' for function 'nullIn' |
Not this PR (a known upstream bug family) |
| GrypeScanServer / Grype Scan | Job ran for 6h5m and hit the runner time limit | Infra |
| DCO | Commits have no Signed-off-by |
Process. Every antalya-26.6 backport fails it (#2900, #2848) |
Stress test (amd_debug): 3 of 4 sub-checks failed
Received signal 6while running a plain negativeLIMIT BYovernumbers():The query comes from the negative-(query: SELECT ... FROM numbers(15) AS `__table1` ORDER BY `__table1`.`number` % 3 ASC, `__table1`.`number` ASC LIMIT _CAST(-3, 'Int64') BY `__table1`.`number` % 3) Received signal Aborted (6) ... DB::executeQueryImpl ... executeQuery.cpp:2247 ... TCPHandler::runImplLIMIT BYtests (03837_negative_limit_by,04108_negative_limit_by_edge_cases, …). It reads no storage and writes nothing, soPartitionedSinkis not involved. The abort happens insideexecuteQueryImpland noLogical errorline was logged before it, which points to a debug-only consistency assertion in the negativeLIMIT BYfeature.Cannot start clickhouse-serverafter the stress run:This table was left behind byLoad job 'load table test_ycrjgnmt.outer_alias_05213' failed: Code: 36. DB::Exception: Alias table cannot refer to another Alias table ... ENGINE = Alias('test_ycrjgnmt', 'inner_alias_05213')05213_storage_alias_chain_join_snapshot.sql. Upstream tracks it as Server cannot start after restart with anAlias→Aliaschain in metadata:CREATE-time validation enforced atATTACHbreaks stress tests ClickHouse/ClickHouse#116207: the check that blocks anAlias→Aliaschain onCREATEis also enforced onATTACH, so the server cannot start after a restart. That issue is closed upstream, but the fix doesn't seem to be in antalya-26.6 yet. Either way it has nothing to do with this PR.- "Check failed" is the summary of the two failures above.
Stress test (amd_asan_ubsan): 1 of 5 sub-checks failed
<Fatal> : Logical error: 'Not-ready Set is passed as the second argument for function 'nullIn''.
(query: SELECT count() = 33333 FROM null_in_subquery WHERE i IN (SELECT i FROM null_in_subquery WHERE dt != 1025)) Received signal Aborted (6)
The query comes from 01231_operator_null_in_2.sql (a SELECT with IN (subquery)). This is the open upstream "Not-ready Set" analyzer/planner bug family: ClickHouse#118722, ClickHouse#72739, ClickHouse#121039. It has nothing to do with partitioned writes.
GrypeScanServer / Grype Scan
The job ran for 6h5m, which is the GitHub Actions job time limit. That is a hang or timeout, not a reported vulnerability. On the sibling backports #2900 and #2848 the same GrypeScanServer job passed in 5–9 minutes. Safe to re-run.
DCO
The commits come from the backport bot (b554bdd, d551664) and a merge (02439a6), and none have Signed-off-by. Other antalya-26.6 backports (#2900, #2848) fail DCO the same way. Normally this is ignored or overridden by a maintainer. Fixing it would mean rewriting history, which needs a force-push, so I didn't touch it.
Caveat and suggested next steps
- On the sibling PRs Antalya 26.6 Backport of #131001 - Reuse the GCM cipher context between rows in encrypt/decrypt #2900, Antalya 26.6 Backport of #131154 - Reject out-of-range
Decimalscale when reading aField#2848 and Antalya 26.6 - backport 120107 - Do not re-pull integration images the job already prefetched #2927 the stress jobs passed. The stress suite runs random tests in random order, so I'm judging these by what crashed, not by comparing runs. All three crashes are in query paths this PR doesn't touch (negativeLIMIT BY,INsets, attachingAliastables). - No code fix is needed in this PR. Re-run
Stress test (amd_debug),Stress test (amd_asan_ubsan)and the Grype scan. - If the
Alias→Aliasstartup failure keeps coming back on antalya-26.6, it's worth backporting the upstream fix for Server cannot start after restart with anAlias→Aliaschain in metadata:CREATE-time validation enforced atATTACHbreaks stress tests ClickHouse/ClickHouse#116207 separately. I can do that if you want.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fixed
PARTITION BYoffile,urland object storage writes (table functions and table engines) evaluating stateful and non-deterministic functions anew for every block since 26.1:PARTITION BY intDiv(rowNumberInAllBlocks(), N)restarted its counter on every block and wrote rows into the wrong files, andnow64()produced one file per block instead of one perINSERT. Closes ClickHouse#128211. (ClickHouse#128217 by @groeneai)Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: